Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: xtifr Date: Apr 20, 2005 8:59am
Forum: etree Subject: warning to uploaders re: search engine

Just a heads-up for uploaders. Sometimes (for no obvious reason) a show will fail to get picked up by the search engine. The show will still be visible if you start at the "Browse Bands" link on the Archive's front page, but it cannot be searched, and will not appear (for example) on the "most recently added shows" list.

To check for this, and see if it's happened to your upload, wait until the show has been made public, then go to the "Browse bands in the Live Music Archive" link, click on the band name, then click on the tiny link marked "(other recent additions...)" which appears right after the Most Recent Show link. If your show does not appear, you're a victim of the bug.

To fix the problem, go to the show record (you'll have to browse by date, since you can't search for it), and then click on "[edit]" and then "update". This will cause the search engine to (re)index the show, and everything should be fine from then on.

To summarize: ALWAYS check your shows AFTER they've been made public, and, if they don't show up in the search engine, edit and update.

Reply to this post
Reply [edit]

Poster: bstover Date: Apr 21, 2005 7:15am
Forum: etree Subject: Re: warning to uploaders re: search engine

Hmmm... that procedure did not work with this recording, which is still in search engine never-never land:

http://www.archive.org/audio/etree-details-db.php?id=24563

Reply to this post
Reply [edit]

Poster: xtifr Date: Apr 21, 2005 12:42pm
Forum: etree Subject: more search engine problems

Hmm, indeed. It's always worked for me in the past, and has worked for several other people. However, in a completely unrelated incident, a fellow I know updated several of his show's records today (to correct the venue name), and two of those shows, which *had* been in the search engine, are no longer searchable!

It seems that updating the record can trigger the bug as well as fixing it. Either that, or the search engine has just had a catastrophic meltdown today. I'd try again, and if that doesn't fix the problem (after waiting a while as Tyler suggested), then I'm not sure what to suggest.

Reply to this post
Reply [edit]

Poster: kmcmurtrie Date: Apr 22, 2005 1:45am
Forum: etree Subject: Re: more search engine problems

I just noticed on main audio page there is a total of 22,442 shows, but on most recently added page it was 21,776, a difference of 666. Coincidence or demonic possession of the search engine?

Reply to this post
Reply [edit]

Poster: xtifr Date: Apr 22, 2005 4:26pm
Forum: etree Subject: Re: more search engine problems

I don't know - how many heavy metal bands are there on the Archive? :)

Man, that's a lot of shows that are completely unsearchable! That's like 7 or 8% of the entire archive that you can't find unless you know in advance exactly what you want. That's a drag!

Reply to this post
Reply [edit]

Poster: kmcmurtrie Date: Apr 23, 2005 12:35am
Forum: etree Subject: Re: more search engine problems

More like 3%, but still a very serious problem.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffsimon c Date: Apr 23, 2005 1:59am
Forum: etree Subject: Re: more search engine problems

On this subject, we have an easy way to (manually) re-add things to the search engine now, so I'll speak to the etree admins about this - if we could get a list of the missing item names together, it would be easy to add 'em, I think.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Apr 23, 2005 4:27am
Forum: etree Subject: Re: more search engine problems

As Christopher points out, finding what's missing is the hard part!

Here's one possible way to go about it: In one tab, check browse page and note number of shows for the band (ex: Radiators, 313). In another tab, do a search of LMA and then filter to creator (Radiators, 310). In a case where there's a number mismatch (3 missing here), maybe sort search results by date and visually compare item listings between the 2 tabs (16 listed in browse for 2005; 16 2005's listed in search; that year at least is OK).

Main problem with that idea: Tedious and time consuming esp for many-item or many-year bands, like Radiators.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Apr 23, 2005 6:22am
Forum: etree Subject: Re: more search engine problems

I picked up the 3 missing Radiators while testing this way:
http://www.archive.org/audio/etree-details-db.php?id=24569
http://www.archive.org/audio/etree-details-db.php?id=24587
http://www.archive.org/audio/etree-details-db.php?id=20313

Trail of the Dead:
http://www.archive.org/audio/etree-details-db.php?id=22702

6gig:
http://www.archive.org/audio/etree-details-db.php?id=21732

Warren Zevon:
http://www.archive.org/audio/etree-details-db.php?id=24563
http://www.archive.org/audio/etree-details-db.php?id=24596
BTW the nonstandard naming of this WZ item
http://www.archive.org/audio/etree-details-db.php?id=24451
makes it show up out of date order in the search listings. Similar quirks in long lists would make this kind of search even harder to do. That's another reason for people to follow the Naming Standards:
http://wiki.etree.org/index.php?page=NamingStandards


This post was modified by Diana Hamilton on 2005-04-23 13:22:01

Reply to this post
Reply [edit]

Poster: bstover Date: Apr 23, 2005 12:03pm
Forum: etree Subject: Re: more search engine problems


> Warren Zevon:
> http://www.archive.org/audio/etree-details-db.php?id=24563
> http://www.archive.org/audio/etree-details-db.php?id=24596


Those are the ones, Diana. They're the two most recent uploads. Will they be manually added to the search engine at some point? I would be happy to do that if instructed how. Would then be pleased to keep the tiny WZ corner of the LMA shipshape.

Thanks for your diligence!

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffsimon c Date: Apr 23, 2005 12:46pm
Forum: etree Subject: All added (was Re: problems)

I just added all of Diana's items to the search engine - tell me if they're not showing up properly now.

Reply to this post
Reply [edit]

Poster: xtifr Date: Apr 24, 2005 6:23am
Forum: etree Subject: Re: All added (was Re: problems)

Hmm, as far as the Rads go, I was able to find the 11/19/04 show with the search engine just a few days ago. Either Diana got that one wrong, or it dropped off the map while no one was looking. The only missing Rads show from 2004 I know of (and it's still missing) is:

http://www.archive.org/audio/etree-details-db.php?id=20554

If you fix that one, then the Radiators (at least) will have all their shows searchable. (Yes, the other two are definitely fixed). Note: if you fix it, you can close the error report I filed a few days ago. :)

edit: addendum: the fact that Diana either missed one show or linked to the wrong show just serves to emphasize, I think, how labor-intensive and error prone the manual approach is.

This post was modified by xtifr on 2005-04-24 13:21:01

This post was modified by xtifr on 2005-04-24 13:23:53

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffsimon c Date: Apr 24, 2005 6:30am
Forum: etree Subject: Re: All added (was Re: problems)

Agreed - this is not a sensible way to do things in the longer-term. But in the short-term, if people happen to spot missing items, esp. if there's only one or two missing from their own 'specialized' area of expertise, I can add 'em.

BTW, that particular Radiators show seemed to be missing XML, which was perhaps why it couldn't be added to the search engine - I updated it and then it worked.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Apr 24, 2005 9:35am
Forum: etree Subject: Re: All added (was Re: problems)

Longer-tern, would the move to the petabox make this batch of missing go away (I'm guessing due to a reindexing)? And/or create a whole new set? ;)

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffsimon c Date: Apr 24, 2005 11:45am
Forum: etree Subject: Re: All added (was Re: problems)

Not an official answer, but I believe that the search engine's adding quirks are probably a low priority project compared to things like data security and moving to the petabox (which IMHO won't make any difference to this particular problem) - that's why I suggest some basic manual cleaning up.

[Not that it's not going to get fixed eventually, I hope!]

Reply to this post
Reply [edit]

Poster: kmcmurtrie Date: Nov 10, 2005 10:27pm
Forum: etree Subject: Re: All added (was Re: problems)

Almost 6 months later and the search engine is now missing 1,187 shows - over 4%. Perhaps this problem could be given a higher priority? It seems that a critical function of an archive this size would be a reliable search. But hey, everything else you good folks are doing is great!

Reply to this post
Reply [edit]

Poster: bstover Date: Apr 25, 2005 5:13am
Forum: etree Subject: Re: All added (was Re: problems)

Thanks, simon c,

The referenced WZ recordings are now searchable.

Cheers,
B.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffTyler Date: Apr 21, 2005 8:51am
Forum: etree Subject: Re: warning to uploaders re: search engine

also remember that the search engine list is cached.. it may need a specific amount of time as well to be 're-uped' into the system's list and to be 'seen'.

Reply to this post
Reply [edit]

Poster: bstover Date: Apr 21, 2005 12:45pm
Forum: etree Subject: Re: warning to uploaders re: search engine

Well I dunno. I updated the "Taped by" and Transferred by" fields for the recording more than four hours ago and they were immediately visible. However, expanding the recording info on the contributions page indicates that the last update was some 13 hours ago and was done by the contribution engine:

task_id: 107084
item_identifier: wz1983-01-08.sbeok.flac16
mediatype: audio
collection: etree
progress: MD5 checksum verification passed. Copying and deleting (i.e. moving) worked fine. Files now reside on audio43.archive.org
contributor: bstover@arkapple.com
server_from: audio-uploads.archive.org
updater: Contribution Engine
wait_admin: 0
wait_user: 0
wait_curate: 0
update_only: 0
working_on:
updated: 2005-04-21 06:38:25 (13.00 hr)
priority: 2
done: 1


Seems inconsistent.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffBrad Leblanc Date: Apr 24, 2005 3:37am
Forum: etree Subject: Re: warning to uploaders re: search engine

Well I dunno. I updated the "Taped by" and Transferred by" fields for the recording more than four hours ago and they were immediately visible. However, expanding the recording info on the contributions page indicates that the last update was some 13 hours ago and was done by the contribution engine. Seems inconsistent.

Well - you are talking about 3 different processes:

1. Updating the Metadata.
2. Log from the Contribution Engine (that stuff you pasted above).
3. Updating the Search Engine.

The CE log does not change after the item goes into the collection and will not change if you edit the metadata - however, if you are the contributor there is a section on the details page that lets you know the last time the item metadata was edited and by whom - this is under the "Transferred By" field (which is now spelled correctly - thanks Tracey!)

Your recording is now in Search Engine. Updating the metadata "looks" instantaneous, but really takes 5-10 minutes before it gets written to the meta.xml file in the show directory. Adding items to the search engine with the Edit > Update method has taken hours in my experience, but usually works when I check on it the next day.

This post was modified by Brad Leblanc on 2005-04-24 10:37:48